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ABSTRACT 



A surgical navigation system has a computer with a memory 
and display connected to a surgical instrument or pointer and 
position tracking system, so that the location and orientation 
of the pointer are tracked in real time and conveyed to the 
computer. The computer memory is loaded with data from 
an MRI, CT, or other volumetric scan of a patient, and this 
data is utilized to dynamically display 3-dimensional per- 
spective images in real time of the patient's anatomy from 
the viewpoint of the pointer. The images are segmented and 
displayed in color to highlight selected anatomical features 
and to allow the viewer to see beyond obscuring surfaces 
and structures. The displayed image tracks the movement of 
the instrument during surgical procedures. The instrument 
may include an imaging device such as an endoscope or 
ultrasound transducer, and the system displays also the 
image for this device from the same viewpoint, and enables 
the two images to be fused so that a combined image is 
displayed. The system is adapted for easy and convenient 
operating room use during surgical procedures. 

3 Claims, 15 Drawing Sheets 
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METHOD FOR VOLUMETRIC IMAGE 
NAVIGATION 

CROSS REFERENCE TO RELATED 
APPLICATIONS 

This application is a continuation of application Ser. No. 
08/884,289, filed Jun. 27, 1997 now abandoned. 

This application claims the benefit of U.S. Provisional 
Application Ser. No. 60/020,664 filed Jun. 28, 1996. This 
Provisional Application Ser. No. 60/020,664 is hereby incor- 
porated by reference in its entirety into the present applica- 
tion. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention pertains generally to systems and methods 
for generating images of three dimensional objects for 
navigation purposes, and more particularly to systems and 
methods for generating such images in medical and surgical 
applications. 

2. Description of the Background Art 

Precise imaging of portions of the anatomy is an increas- 
ingly important technique in the medical and surgical fields. 
In order to lessen the trauma to a patient caused by invasive 
surgery, techniques have been developed for performing 
surgical procedures within the body through small incisions 
with minimal invasion. These procedures generally require 
the surgeon to operate on portions of the anatomy that are 
not directly visible, or can be seen only with difficulty. 
Furthermore, some parts of the body contain extremely 
complex or small structures and it is necessary to enhance 
the visibility of these structures to enable the surgeon to 
perform more delicate procedures. In addition, planning 
such procedures requires the evaluation of the location and 
orientation of these structures within the body in order to 
determine the optimal surgical trajectory. 

New diagnostic techniques have been developed in recent 
years to obtain images of internal anatomical structures. 
These techniques offer great advantages in comparison with 
the traditional X-ray methods. Newer techniques include 
micro impulse radar (MIR), computer tomography (CT) 
scans, magnetic resonance imaging (MRI), positron emis- 
sion tomography (PET), ultrasound (US) scans, and a vari- 
ety of other techniques. Each of these methods has advan- 
tages and drawbacks in comparison with other techniques. 
For example, the MRI technique is useful for generating 
three-dimensional images, but it is only practical for certain 
types of tissue, while CT scans are useful for generating 
images of other anatomical structures. Ultrasound scanning, 
in contrast, is a relatively rapid procedure; however it is 
limited in its accuracy and signal-to-noise ratio. 

The imaging problem is especially acute in the field of 
neurosurgery, which involves performing delicate surgical 
procedures inside the skull of the patient. The above tech- 
niques have improved the surgeon's ability to locate pre- 
cisely various anatomical features from images of structures 
within the skull. However this has only limited usefulness in 
the operating room setting, since it is necessary to match 
what the surgeon sees on the 2D image with the actual 3D 
patient on the operating table. The neurosurgeon is still 
compelled to rely to a considerable extent on bis or her 
knowledge of human anatomy. 

The stereotactic technique was developed many years ago 
to address this problem. In stereotactic surgery, a frame of 
reference is attached to the patient's head which provides 
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reference points for the diagnostic images. The device 
further includes guides for channeling the surgical tool along 
a desired trajectory to the target lesion within the brain. This 
method is cumbersome and has the drawback that the 

5 surgeon cannot actually see the structures through which the 
trajectory is passing. There is always the risk of damage to 
obstacles in the path of the incision, such as portions of the 
vascular or ventricular system. In essence, with previous 
neurosurgical techniques the surgeon is in the position much 
like that of a captain piloting a vessel traveling in heavy fog 
through waters that have many hazards, such as shoals, 
reefs, outcroppings of rocks, icebergs, etc. Even though the 
captain may have a very good map of these hazards, 
nevertheless there is the constant problem of keeping track 
of the precise location of the vessel on the map. In the same 

15 way, the neurosurgeon having an accurate image scan show- 
ing the structures within the brain must still be able to 
precisely locate where the actual surgical trajectory lies on 
the image in order to navigate successfully to the target 
location. In the operating room setting, it is further necessary 

20 that this correlation can be carried out without interfering 
with the numerous other activities that must be performed by 
the surgeon. 

The navigation problem has been addressed in U.S. Pat. 
No. 5,383,454, issued Jan. 24, 1995 (Bucholz). This patent 

25 describes a system for indicating the position of a surgical 
probe within a head on an image of the head. The system 
utilizes a stereotactic frame to provide reference points, and 
to provide means for measuring the position of the probe tip 
relative to these reference points. This information is con- 

3 q verted into an image by means of a computer. 

U.S. Pat. No. 5,230,623, issued Jul. 27, 1993 (Guthrie), 
discloses an operating pointer whose position can be 
detected and read out on a computer and associated graphics 
display. The pointer can also be used as a "3D mouse" to 

35 enable the surgeon to control the operation of the computer 
without releasing the pointer. 

U.S. Pat. No. 5,617,857, issued Apr. 8, 1997 (Chader et 
al.) sets forth an imaging system and method for interac- 
tively tracking the position of a medical instrument by 

40 means of a position-detecting system. The pointer includes 
small light-emitting diodes (LED), and a stationary array of 
radiation sensors is provided for detecting pulses emitted by 
these LED's and utilizing this information to ascertain 
dynamically the position of the pointer. Reference is made 

45 also to U.S. Pat. No. 5,622,170, issued Apr. 22, 1997 
(Schulz), which describes a similar system connected to a 
computer display for displaying the position of an invasive 
surgical probe relative to a model image of the object being 
probed (such as a brain). 

50 U.S. Pat. No. 5,531,227, issued Jul. 2, 1996 (Schneider) 
explicitly addresses the problem recognized in many other 
references that it is desirable to provide a real time display 
of a surgical probe as it navigates through the brain. This 
patent describes a system for providing images along the 

55 line of sight of the surgeon in a dynamic real-time fashion. 
Id this system the images that are displayed are resliced 
images from a three-dimensional data reconstruction which 
are sections or slices orthogonal to the line of sight, taken at 
various positions along this line specified by the user. Thus, 

60 while the viewpoint for the line of sight is always external 
to the body, the sectional planes that are used to define the 
virtual images may constitute various slices through the 
body chosen by the surgeon. These images may be super- 
imposed on actual images obtained by an image recording 

65 device directed along the line of sight such as a video camera 
attached to the surgeon's head, and the composite images 
may be displayed. 
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The systems described above attempt to address the instrument is moved or rotated, the position tracking system 

navigation problem in various ways, and they all have the continually provides data to the computer indicating the 

common drawback of requiring a certain level of abstract location and orientation of the instrument, and the displayed 

visualization by the surgeon during an operating room image is continually updated to show the structures toward 

procedure. When the surgeon is proceeding through the 5 which the instrument is pointing. 

brain toward a target tumor or lesion, it is desirable to be i n addition, for probes or instruments being used that are 

fully aware of all of the structures around the surgical capable themselves of generating images, such as ultrasound 

trajectory. With previous systems the displays that are pre- probes, endoscopes, or surgical microscopes, the system 

sented do not provide all of this information in a single provides means for integrating these images with those 

convenient real-time display, and they require the viewer to 10 generated from the scan data. The software enables the user 

piece together and re-orient the displayed information to to overlay the "actual images" generated by these instru- 

obtain a mental picture of the surrounding structures. These ments with the "virtual images" generated from the scan 

are serious practical disadvantages in an operating room data. 

setting. What is absent from previous systems is a 3D h {s ao o5ject of lhis inveDtion l0 provide a system and 

display that shows, in a real-time view, the various structures is method for genC rating an image in three dimensional per- 

looking ahead from the surgical probe along a line of sight spe ctive of anatomical structures encountered by a surgeon 

into the brain in three and two dimensions, including struc- during the performance of surgical procedures, 

tures hidden by other features. A object of ^ invention is to provide a system 

SUMMARY OF THE INVENTION 20 a ° d m f*° d f ° f g eDeratia S ^ h an with user- 

controlled means for varying the location and orientation of 

The present invention provides an improved system and the viewpoint corresponding to the image. 

method for displaying 3D images of anatomical structures in Another object of this invention is to provide a system and 

real time during surgery to enable the surgeon to navigate method for generating such an image with user-controlled 

through these structures during the performance of surgical means for varying the opacity of structures and surfaces in 

procedures. This system is also useful in planning of surgical the viewed region of interest, so that the displayed image 

procedures. The system includes a computer with a display shows structures and features that would be otherwise 

and input devices such as a keyboard and mouse. The system hidden in a normal view. 

also includes a position tracking system that is connected Y et another object of this invention is to provide a system 

both to the computer and also to the surgical probes or other 3o and me thod for generating such an image with a viewpoint 

instruments that are used by the surgeon. The position located at the tip of the instrument being used by the surgeon 

tracking system provides continual real time data to the m the direcUon along me longitudinal axis of the instrument. 

computer indicating the location and orientation of the 0# . n t , , . , f( . . . • ■ 

f . ™ r . . . . Still another objector this invention is to provide a system 

surgical instrument in use. The computer further includes a an(J method for tin such an ^ m rea , tim wch 

memory contain^ patient data produced I by , maging scans 35 ^ ^ * d { t0 the 

such as CT or MRI scans, from which 2-dimens.onal and itioQ of ^ bein ^ b the s 

3-dimensional images of the anatomical structure may be , r _ . ,. . -7 

generated. Means are provided for registration of these Yet a further object of this invenUon iS to provide a system 

images with respect to the patient. and method for companng and combining such an image 

J~ . v . . with the image produced by an image -generating instrument 

The computer memory is further provided with programs ^ bei ^ b tfae surgeon 

that control the generation of these anatomical images. ..... 

™ • t j c e t c *u These and other objects, advantages, characteristics and 

These programs include software for segmentation of the _ r . ■ * . . * . 

• . m »t • • c . a features of the mvention may be better understood by 

scan unages to identify various types of structures and .. . r „ . , - . • L L j •, i 

„ # , ' . e . • examining the following drawings together with the detailed 

tissues, as well as the reconstruction of 2D and 3D images , • ■ * . r j l j« 

c *tl , , tv.- a 11 «l - . u description of the preferred embodiments, 

from the scan data. This software allows these images to be 45 r r 

displayed with various magnifications and orientations, and BRIEF DESCRIPTION OF THE DRAWINGS 

with various sectional views produced by slice planes in « . , . , r , 

various locations and orientalions, all controlled by the t u schematic perspective drawing of the apparatus 

of the present invention in operating room use during the 

surgeon. performance of neurosurgical procedures. 

This image-generating software has the important feature 50 - . . t . . c 4l _ 

. , & & - T> 6 iL 4 ... r lL FIG. 2 is a schematic block diagram of the computer 

that it produces 3D images that are perspective views of the . . . . . . . , T , . r . 

r . . e . . iV j r system and optical tracking system of the present invention, 

anatomical structures, with user-controlled means for vary- m - * * . 

ing the viewing orientation and location, and also varying FIG. 3 is a schematic block diagram of the navigation 

the displayed transparency or opacity of various types of P rotoco1 P^-operative data that is followed in carrying 

tissues, structures, and surfaces in the viewed region of 55 0Ut lhe method of the prcSCnt mveDtl0a - 

interest. This enables the user to effectively "see through" FIG. 4 is a schematic block diagram of the navigation 

surfaces and structures in the line of sight of the image to protocol using ultrasound mtra-operative data that is fol- 

reveal other structures that would otherwise be hidden in lowed in carrying out the method of the present invention, 

that particular view. FIG. 5 is a schematic block diagram of the endoscopic 

Further, the images are generated from the viewpoint of 60 protocol that is followed in carrying out the method of the 

the surgical probe or instrument that is in use, looking from present invention. 

the tip of the instrument along its longitudinal axis. Thus, FIG. 6 is a schematic flow chart of the pre-operative 

when an invasive surgical instrument such as a scalpel or computer program that implements the pre-operative proto- 

forceps is inserted into an incision in the body, the display col of the present invention. 

provides a three dimensional perspective view of anatomical 65 FIG. 7 is a schematic flow chart of the intra-operalive 
structures from a viewpoint inside the body. These images ultrasound computer program that implements the ultra- 
are all generated in real time "on the fly". Thus, as the sound protocol of the present invention. 
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FIG. 8 is a schematic flow chart of the intra -operative Colo., similar to the systems described in U.S. Pat. No. 

endoscope computer program that implements the endo- 5,617,857 (Chader, et al.) and U.S. Pat. No. 5,622,170 

scope protocol of the present invention. (Schulz) discussed previously. However the invention is not 

FIG. 9 is a drawing of a display generated according to the Umited t0 mis Particular OTS, and other position tracking 

present invention, showing axial, coronal, and sagittal views 5 systcnw, «ich as sonic position detecting systems, may also 

of a head, together with a three-dimensional perspective e u ^ ' - . . 

view of the head taken from an exterior viewpoint. , M pirated m FIG. 1, the surgical instrument 109 is 

m „ . , ■ elongated in shape, havmg a longitudinal axis and tip 115 

FIG. 10 is a drawing of a display generated according to imin lQward tfae of iQ{ ^ ^ inslrumem may 

the present invention, showing sectional axial, coronal, and be M endoscope having a ^ field of view 116 lhal fe 

sagittal views of a head, together with a three^hmensional mdicaled by dotted lines in FIG. 1. The instrument shown in 

perspective v ie w of the head taken from an interior view- ^ ¥{ ^ n ^ hdd at a posilion extemal lQ ^ head 

P oml * If an incision 118 has been made into the skull, the instru- 

FIG. 11a is a drawing of a plastic model of a human skull meDl may ^ msert ed through the incision; this alternative 

and a surgical probe that has been used to demonstrate the ^ position is shown by dotted lines in FIG. 1. In both positions 

present invention. the instrument is held so that there is an unobstructed line of 

FIG. lib is another drawing of the model skull of FIG. sight between the LED's 110, 111 and the sensing unit 105. 

11a, with the top of the skull removed to show model In endoscopic and other optical viewing applications, the 

internal structures for demonstration purposes. instrument may include a laser targeting system (not shown 

FIG. 12 is a simplified reproduction of two displays 2 o m me drawings) to illuminate and highlight the region under 

produced by the present invention for the model skull shown examination. 

in FIGS, llfl, lib. FIG. 2 shows a schematic block diagram of the computer 

FIG, 13 is a simplified reproduction of two further dis- system connected to the position tracking system. The 

plays of the invention for the skull in FIGS, llfl, lib. computer 101 includes a central processing unit (CPU) 201 

FIG, 14 is a reproduction of a composite display produced 25 communicative with a memory 202, the video display 102, 

by the present invention for an actual human head. keyboard and mouse 103, 104, optical sensors 106-108, and 

LED's 110,111 (FIG. 1) mounted on the surgical instrument 
DETAILED DESCRIPTION OF THE 109. The computer memory contains software means for 
PREFERRED EMBODIMENTS operating and controlling the position tracking system. In an 
FIG. 1 shows the apparatus of the invention as used in 30 alternative preferred embodiment, the OTS components 
performing or planning a neurosurgery operation. In this 105-108 may be connected to and controlled by a separate 
drawing the patient's head 112, has a tumor or lesion 117, computer or controller which is connected to the computer 
which is the target object of the operation. Fiducial markers 101 and provides continual data indicating the position and 
113, 114 are attached to the head to enable registration of orientation of the surgical instrument 109. 
images generated by previously obtained scan data accord- 35 The above apparatus is operated to carry out surgical 
ing to techniques familiar to persons of ordinary skill in the protocols that are illustrated schematically in FIGS. 3-5. 
relevant art. A surgical probe or instrument 109 held by the FIG. 3 is a schematic block diagram of the protocol for 
surgeon is directed toward the tissues of interest. A computer handling pre-operative data ("pre-op protocol") to generate 
101 is connected to user input devices including a keyboard images during surgery according to the present invention. It 
103 and mouse 104, and a video display device 102 which 40 is assumed that three-dimensional image data of the 
is preferably a color monitor. The display device 102 is patient's head have been previously obtained from one or 
located such that it can be easily viewed by the surgeon more of the techniques that are known to persons of ordinary 
during an operation, and the user input devices 103 and 104 skill in the medical imaging arts. Preferably these data are 
are placed within easy reach to facilitate use during the acquired from CT, MIR and/or MRI scan techniques to 
surgery. The apparatus further includes a position tracking 45 provide images with improved accuracy and detail, corn- 
system, which is preferably an optical tracking system pared to ultrasound scan data for example. The scan data are 
(hereafter "OTS") having a sensing unit 105 mounted over- loaded and stored 301 into the computer memory 202 
head in view of the operating table scene, and at least two through additional input means such as disk drives or tape 
light emitting diodes (LED's) 110, 111 mounted on the drives, not shown in the drawings. 

surgical instrument 109. These LED's preferably emit con- 50 The patient data is registered 302 according to one of the 

tinuous streams of pulsed infrared signals which are sensed generally known techniques. This procedure may be either a 

by a plurality of infrared sensors 106, 107, 108 mounted in three-dimensional registration of the entire data set, or a 

the sensing unit 105 in view of the surgical instrument 109. slice -by-slice sequence of two-dimensional registrations. 

The instrument 109 and the sensing unit 105 are both Following the three-dimensional registration, the image is 

connected to the computer 101, which controls the timing 55 reconstructed 303 in memory, using volumetric or surface 

and synchronization of the pulse emissions by the LED's rendering to produce an array of 3-dimensional voxel data, 

and the recording and processing of the infrared signals Segmentation 304 is then carried out on these data to 

received by the detectors 106-108. The OTS further distinguish various anatomical features, such as different 

includes software for processing these signals to generate types of material in the head (bone, brain tissue, vascular and 

data indicating the location and orientation of the instrument 60 ventricular structures, etc.) and the location of surfaces, 

109. The OTS generates the position detecting data on a real using one or more of known segmentation techniques, 

time continuous basis, so that as the surgical instrument 109 Preferably the segmentation process includes assigning dif- 

is moved, its position and orientation are continually tracked ferent display colors to different types of structures to 

and recorded by the sensing unit 105 in the computer 101. facilitate their identification and distinction in a color video 

The OTS may be preferably of the type known as the 65 display. For example, the vascular system may be displayed 

"FlashPoinl 3-D Optical Localizer", which is commercially in red, the ventricular system may be shown in blue, bones 

available from Image Guided Technologies of Boulder, may be colored brown, and so on. In a preferred embodiment 
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these assignments may be varied by the user by means of the surgical instrument 109. This protocol is also applicable for 

keyboard 103 or mouse 104. Also in a preferred embodiment procedures utilizing a surgical microscope in place of the 

the display opacities may be varied by the user by means of endoscope. Image data is acquired 501, using a CCD camera 

the keyboard 103, mouse 104, or other input device (such as or other known technique, representing a 2-dimensional 

a voice-activated device) to further facilitate their identifi- 5 m a P lane orthogonal to the line of sight of the 

cation and distinction of hidden or obstructed features in the endoscope or microscope, which in this context is the 

video display. In an alternative protocol in which longitudinal axis of the surgical instrument 109. The loca- 

2Hdimensional registration is carried out, segmentation 309 tion and orientation of the instrument is tracked and deter- 

can be done for each 2Hdimensional image sample, and the mmed 502 b y me 0TS ' and analog-to-digital ("A/D") con- 

3<iimensional data are then reconstructed 310 from the 10 version 503 is carried out on the data. The location of the 

segmented data slices. This alternative protocol is shown by viewpoint is determined 504 from the OTS data, and the 

dotted lines in the Figure endoscope or microscope image data is manipulated 505 to 

_ r . ... , JL r generate the desired image 506 for display. These intra-op 

Referring still to FIG. 3, the next phase of the pre-op r - . _ AO .f? , * t j . 

. . * , iL \ 4 . • * unages are fused 508 with the pre-op images generated by 

protocol is to determine the location and orientation of the t . ° . ■ in j *u ■ 

*\ . ™* , _. * iLJ *i j t, . n e the pre-op protocol 311, and the composite images are 

view vector 305 to define the image to be displayed. This 15 ,, r , . „ tl _ . r „ . . , 

. . ■ • j l ■ *u r^rj^ . ' . 4 . further displayed. Finally, the OTS is continually strobed 

view vector is obtained by querying the OTS to ascertain the cevj ' . : . ; f , . 

. . * * * ** c *i_ • i • * * 507, and the ultrasound unages are constantly refreshed, 

current location and orientation of the surgical instrument ' , . , \ . 

109. With this information, the three-dimensional scan data The foregomg protocols are trnplernented by program 

is then manipulated 306 to position and orient the resulting stored . ,n ,he u mem ° r ? 202 of the computer 101. 

three^imensional perspective view and to define cutting 20 FIG. 6 u a schematic block d.agram of a flowchart for a 

planes and reference markers in the displayed image indi- progra * "l? 1 ""Pimento the P re "°P P ro,oco1 - ^ P«W«"J 

eating and clarifying this view. The manipulated three- sUrts 60 } by "using the computer to receive and load 602 

dimensional perspective image is then displayed 307 on the Piously obtatned scan data for the patient such as MRI 

video display 102. In addition, other two-dimensional °' CT data : The computer further reads data from the OTS 

images, such as 2D sectional views for any cutting planes, 25 603 to register the scanned patient data 604. For 3D volu- 

are preferably also displayed along with the 3D perspective metnc ™ d ™Z' «^ uied ? ala 18 used 10 reconsl ™ c ' 

display for purposes of elucidation. ! ma S e da ! a 605 ,n lh [ ee dimensions, and segmentation 606 

„ , , . ... is earned out on this reconstruction. In an alternative 

• Fl ? a " y V the P^"° P pr0,0 °° 18 3 "^ol l0 °? embodiment, shown by dotted lines in the Figure, segmen- 

in which the OTS is repeatedly queried 308 for changes in talion h cafried out on 2D slices 615> and these 

the location of the v.ew vector corresponding to changes ui slices m (hen reconstructed mt0 lhe M 3D ^ dm , 

the position and onentation of the surgical instrument 109. ~, . r *u t u _j mi 

™ . j . 11 u • j * j The program next reads input data from the keyboard 103 

Thus the displayed images are continually being updated r r nA t . r , . t r , , ' _ . f 

... . , j j * f • j ■ * or mouse 104 to enable the user to select a field of view for 

during the surgical procedure, and the resulting displays are . % £M ~, . , t . . , , , , 

. .i c u j * i ^ • j.i image displays 607. The image data is then manipulated and 

constantly refreshed in real time. The image data are also , & ~ K , i no t . ~. . j • i -.u 

j L cc j j * . i r t iL lit 35 transformed 608 to generate the requested view, along with 

stored or buffered and made available for further use 311 ^ m * 

accordmg to subsequent protocols. J othef Qp{iom {Q ^ ^ by ^ progfam , n 

The surgical instrument 109 may include an ultrasound addition, the user may request a 3D display of the entire 

transducer located at the tip 115, which itself scans and headj loget her with a superimposed cone showing the field 

detects ultrasound imaging data when placed in contact with ^ of view for an endoscope, microscope, ultrasound 

the patient's head. FIG. 4 is a schematic block diagram transducer, or other viewing device being used during the 

showing the intra-operative ("intra-op") ultrasound ("US") surgery. The resulting manipulated image is then displayed 

protocol for handling the US image data during surgery. 609 pre f erab i y m coi or on the video display 102. The 

Typically the ultrasound transducer is a phased focusing computer next reads the OTS data 610 and determines 611 

array which .generates data from a planar fan-shaped sector 45 wnether tQe surgical instrument has moved. If so, program 

of the anatomical region of interest, where the central axis control retums to the ^cUon 0 f a new field of view 607 and 

of the transducer lies in the plane of the scan sector which, tne successive operations 608-610 shown in FIG. 6. If the 

in this context, is collinear with the longitudinal axis of the position of the instrument has not changed, the displayed 

surgical instrument 109. By rotating the instrument and image is stored 612, refreshing any previously stored display 

transducer about this axis, US scan data is collected and 5Q program further looks for requests from the user 

stored 401 for a cone-shaped volume in the region of 61 3 whether to discontinue operation, and it there are no 

interest. This cone defines the "field of view" of the trans- such reques ts, the operations 611 and 612 are repeated. Thus 

ducer scan. me computer remains in a loop of operations until the user 

The location and orientation of the transducer is tracked requests termination 614. 

and determined 402 by the OTS, and the US data is used to 55 FIG. 7 is a schematic block diagram of a flow chart for a 

reconstruct 403 three-dimensional intra-op image data for program that implements the ultrasound intra-op protocol, 

the region of interest. This data is manipulated 404 in a way The program starts 701 by causing the computer to receive 

analogous to the manipulation 306 of the pre-op data, and anc j i oa d the data from a US transducer at the tip 115 of the 

then used to generate three-dimensional images 405, surgical instrument 109. Such data is produced normally 

together with any desired corresponding two-dimensional 60 using polar or spherical coordinates to specify locations in 

images of the ultrasound data. These intra-op images are the region of interest, and the program converts 703 this data 

fused 406 with the pre^p image generated by the pre-op preferably to Cartesian coordinates. Next, OTS data is read 

protocol 311, and the composite images are further dis- 704 t 0 determine the position and orientation of the surgical 

played. Finally, the OTS is continually strobed 407, and the instrument 109, and US data from the aggregation of aligned 

ultrasound images are constantly refreshed. 65 data slices is utilized to reconstruct 3D image data 705 

FIG. 5 is a schematic block diagram of the intra-op representing the US scan data. This image data is manipu- 

protocol in which an endoscope is placed at the tip 115 of the lated and transformed 706 by the program in a manner 
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similar to the manipulation 608 of the pre-op data 608, and Referring now to FIG. 9, the drawing shows a highly 

the resulting image is displayed 707. simplified sketch of a three-dimensional image display 901 

Similarly to the pre-op program shown in FIG. 6, the OTS obtained by the above system with the surgical probe 109 of 

is queried 709 to determine whether the surgical instrument FIG. 1 in the position illustrated, pointing toward the target 

has moved 713, and if so a new US display image is 5 lesion or tumor 117 inside the patient's head 112. The 

constructed. In a preferred embodiment, the program queries display 901 is a perspective view from the tip 115 of the 

the user 716 whether to carry out another US scan of the probe 109 in FIG. 1. This display is continuously refreshed, 

region of interest. If so, program control returns to the so that as the probe 109 is moved the displayed image 901 

operation 702 in FIG. 7 and fresh US data is obtained by the immediately changes. It will be noted that, although the 

US transducer. If another scan is not requested 716, the 1£J probe 109 in FIG. 1 is shown entirely outside the patient's 

program returns to operation 705 and a new 3D image is head, the display 901 shows internal anatomical structures 

reconstructed from the present US scan data. such as the brain and the target lesion 117. With the present 

If the OTS query 709 determines that the surgical instru- system, the display characteristics can be adjusted in real 

ment has not moved since the last query, the US image is time to emphasize or de-emphasize the internal structures, 

fused 710 with the pre-op image obtained by the program 15 These structures may be distinguished by displays with 

shown in FIG. 6, and the combined image is displayed 711. different colors for different types of material. Also, the 

The OTS is again queried 712 to determine 713 whether the display opacity of the skin, skull, and brain tissue may be 

surgical instrument has moved. If so, the program returns to reduced to provide or emphasize further structural details 

the new scan user query 716. Otherwise the program further regarding the target lesion 117. In short, the display 901 

looks for requests from the user 714 whether to discontinue 20 effectively equips the surgeon with "X-ray eyes" to look at 

operation, and if there are no such requests, the operation hidden structures through obstructing surfaces and objects. 

713 is repeated. Thus the computer remains in a loop of With this display, the entire internal structure of the head 

operations until the user requests termination 715, similarly may be examined and studied to plan a surgical trajectory 

to the pre-op program of FIG. 6. before any incision is made. Furthermore, if the surgical 

The endoscope/microscope intra-op protocol is imple- 2 5 instrument 109 in FIG. 1 is a scalpel, the display 901 allows 

mented preferably by the endoscope intra-op program hav- the surgeon to see any structures immediately behind a 

ing a flow chart shown in schematic block diagram form in surface prior to the first incision. FIG. 9 shows also the 

FIG. 8. Upon starting 801, the program causes the computer conventional axial 902, coronal 903 and sagittal 904 2D 

to receive and load image data from the endoscope 802. This displays for purposes of further clarification and elucidation 

data is digitized 803 and preferably displayed 804 on the 30 of the region under examination. 

video display 102. The OTS is queried 805 to receive When the surgical instrument 109 is an endoscope or US 

information determining the location and orientation of the transducer, the field of view 116 is also indicated in the 

endoscope 806. Using this information, the pre-op data display 901 by the quasi-circular image 905 indicating the 

obtained by the pre-op program illustrated in FIG. 6 is intersection of the conical field of view 116 with the surface 

retrieved 807, and utilized to reconstruct a 3-dimensional 35 of the skin viewed by the endoscope 109. This conical field 

virtual image 808 from the viewpoint of the endoscope. This of view is also superimposed, for completeness, in the 2D 

image is displayed 809, in a manner similar to the 3D display displays 902-904. In a preferred embodiment, displays are 

of images by the pre-op program illustrated in FIG. 6. This also presented showing the actual image seen by the endo- 

image is fused 810 with the endoscope image displayed in scope in the field of view 905, and the 3D perspective image 

operation 804, and the combined image is also displayed 40 for the same region in the field of view 905; these auxiliary 

811. The OTS is then strobed 812 to determine 813 whether displays are not shown in the drawings. Similar auxiliary 

the endoscope has moved since the last query, and if so, displays are preferably included when the instrument 109 is 

program control returns to the operation 802 which refreshes an ultrasound transducer. 

the image data received by the endoscope. Otherwise the After an incision 118 has been made in the patient's head, 

program further looks for requests from the user 814 45 the endoscope may be inserted to provide an internal view 

whether to discontinue operation, and if there are no such of the target anatomy. Referring now to FIG. 10, the drawing 

requests, the operation 813 is repeated. Thus the computer shows a highly simplified sketch of a three-dimensional 

remains in a loop of operations until the user requests image display 1001 obtained by the above system with the 

termination 815, similarly to the pre-op and intra-op pro- endoscope 109 of FIG. 1 in the alternative position shown by 

grams of FIGS. 6 and 7. 50 the dotted lines, pointing toward the target lesion or tumor 

The foregoing program modules may be designed 117. The display 1001 has been manipulated to provide a 
independently, and they can be configured also to run three-dimensional sectional view with a cutting plane pass- 
independently. Thus, the pre-op program may be completed, ing through the tip 115 of the endoscope 109 and orthogonal 
followed by running of either or both of the intra-op pro- to its axis. Again, the endoscope field of view 905 is 
grams. Preferably, however, these programs operate in par- 55 indicated in the display, and in a preferred embodiment 
allel during surgery so that the pre-op data images and auxiliary displays are also presented showing the actual 
intra-op data images are all continually refreshed as the image seen by the endoscope in the field of view 905, and 
operation proceeds. Known methods for parallel execution the 3D perspective image for the same region in the field of 
of programs may be utilized to accomplish this result. view 905; these auxiliary displays are also not shown in FIG. 

The above programs are carried out preferably on a 60 10. This Figure further preferably includes also the conven- 

computer 101 that is adapted for computer graphics appli- tional axial 1002, coronal 1003 and sagittal 1004 2D dis- 

cations. Suitable computers for these programs are commer- plays for purposes of further clarification and elucidation, 

cially available from Silicon Graphics, Inc. of Mountain FIGS. 11a, lib, 12 and 13 illustrate further the three - 

View, Calif. Graphics software modules for most of the dimensional displays that are produced by a preferred 

individual image processing operations in the above pro- 65 embodiment of the present invention. Referring to FIGS, 

grams are also available from Silicon Graphics, Inc. as well 11a, 116, a plastic model of a skull has been fabricated 

as other sources. having a base portion 1102 and a removable top portion 
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1101. These Figures show the model skull 1101, 1102 resting 
on a stand 1106. FIG. 11a also shows a pointer 1104 with 
LED's 1101 connected to an OTS (not shown in the 
drawing) that has been used to generate displays according 
to the invention. A plurality of holes 1103 in the top portion 5 
1101 are provided, which allow the pointer 1104 to be 
extended into the interior of the skull. FIG. lib shows the 
skull with the top portion 1103 removed. A plastic model of 
internal structures 1107 is fabricated inside the skull; these 
internal structures are easily recognizable geometric solids, 10 
as illustrated in the Figure. 

The skull of FIGS. 11a, 116 has been scanned to generate 
"pre-op" image data, which has been utilized to produce the 
displays shown in FIGS. 12, 13. FIG. 12 is a composite of 
two displays 1201, 1202 of the skull with the pointer 1104 35 
directed toward the skull from a top center external location, 
similar to the location and orientation of the pointer shown 
in FIG. 1. The display 1201 is a three-dimensional perspec- 
tive view from this pointer location. The display 1202 is the 
same view, but with the display opacity of the skull material 20 
reduced. This reduced opacity makes the internal structure 
1107 clearly visible, as shown in the Figure. During actual 
use, the system enables the surgeon to vary this opacity in 
real time to adjust the image so that both the skull structure 
and the internal structure are visible in the display in various 25 
proportions. 

It will be noted that the surface contour lines shown in the 
display 1201 are produced by the finite size of the rendering 
layers or voxels. These contour lines may be reduced by 
smoothing the data, or by reducing the sizes of the voxels or 30 
layers. 

FIG. 13 is a composite of two further displays with the 
pointer 1104 moved to extend through one of the openings 
1103. Display 1302 is the view from the tip of the pointer 
inside the skull. Display 1301 is a view of the entire structure 35 
from outside the skull along the pointer axis; in other words, 
display 1302 is substantially a magnification of part of 
display 1301. Display 1301 shows the skull with a portion 
cut away by a cutting plane through the tip of the pointer, 
perpendicular to the pointer axis. Both of these displays 40 
clearly illustrate the perspective nature of the three- 
dimensional displays generated by the present invention. 

Finally, FIG. 14 is a simplified composite of displays 
generated by the system for an actual human head. Display 
1401 is a perspective view of the entire head with a cutaway 45 
portion defined by orthogonal cutting planes as shown. This 
display also shows the field of view of an endoscope 
pointing toward the head along the intersection Line of the 
two cutting planes, with the tip of the endoscope at the apex 
of the cone.. Display 1402 shows the two-dimensional sec- 50 
tioaal view produced by the vertical cutting plane, and 
display 1403 shows the corresponding sectional view pro- 
duced by the horizontal cutting plane. Furthermore, the 
images in displays 1402 and 1403 are also transformed 
(rotated and magnified) and superimposed on the three- 55 
dimensional image in display 1401. 

Both of these displays indicate also the intersection of the 
cutting planes with the conical field of view. Display 1404 
is the actual image seen by the endoscope. Display 1405 is 
a virtual perspective view of the endoscope image re con- 60 
structed from scan data by volume rendering in accordance 
with the present invention. Display 1406 is a virtual per- 
spective view of the image from the endoscope viewpoint 
with a narrower field of view, reconstructed from scan data 
by surface rendering in accordance with the present inven- 65 
tion. This display 1406 would be used with a surgical probe 
in planning a surgical trajectory. Display 1407 is a magni- 
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fi cation of 1406 produced at display 1405 at a wider field of 
view, showing the virtual image that would be seen through 
a microscope. Finally, display 1408 is a segmented three- 
dimensional perspective view of the entire head from the 
scan data utilizing surface rendering, and display 1409 is the 
same view with volume rendering. FIG. 14 illustrates the 
rich variety and versatility of the displays that are possible 
with the present system. All of these displays are presented 
to the surgeon in real time, simultaneously, and can be varied 
on line. 

It is apparent from the foregoing description that this 
invention provides improved means for navigating through 
the anatomy during actual surgical procedures. The system 
enables the surgeon to select and adjust the display with the 
same tool that is being utilized to perform the procedure, 
without requiring extra manual operations. Since the dis- 
plays are provided immediately in real time, the imaging 
does not require any interruption of the procedure. In 
addition, the virtual images provided by this system are 
continuously correlated with the images that are obtained 
through conventional means. 

It will be further appreciated by persons of ordinary skill 
in the art that the invention is not limited in its application 
to neurosurgery, or any other kind of surgery or medical 
diagnostic applications. For example, systems implementing 
the invention can be implemented for actual nautical or 
aviation navigation utilizing information from satellites to 
obtain the "pre-op" scan data. The pointing device can be 
implemented by the vessel or aircraft itself, and the video 
display could be replaced by special imaging goggles or 
helmets. 

The foregoing description of the preferred embodiments 
of the invention has been presented solely for purposes of 
illustration and description, and is not exhaustive or limited 
to the precise forms disclosed. Many modifications and 
variations are possible in light of the above teaching. The 
spirit and scope of the invention are to be defined by 
reference to the following claims, along with their full scope 
of equivalents. 

What is claimed is: 

1. A method for facilitating a diagnostic or surgical 
procedure involving a subsurface target site in a patient, 
comprising 

(a) acquiring volumetric scan data of the target site, 

(b) moving a medical instrument to a selected orientation 
with respect to the patient target site; 

(c) determining the orientation of the medical instrument 
with respect to the patient target site; 

(d) selecting a field-of-view angle, 

(e) constructing from the volumetric scan data, a volu- 
metric perspective image of the target site, as seen from 
the orientation of the medical instrument, at the 
selected field-of-view angle, and 

(f) displaying said perspective image to a person manipu- 
lating the instrument, in real time, allowing the person 
to visualize subsurface target structure at the selected 
field-of-view angle, as the medical instrument is moved 
with respect to the patient target site. 

2. The method of claim 1, wherein said medical instru- 
ment is an endoscope operable to generate a visual image 
from the perspective of a distal-end lens. 

3. The method of claim 2, wherein said displaying 
includes simultaneously displaying the image produced by 
the endoscope and the perspective image generated by said 
constructing. 

***** 



05/06/2004, EAST Version: 1.4.1 



